metadata: {
  "num_sequences": 100000,
  "vocab_size": 32000,
  "max_seq_len": 2048,
  "tokenizer": "llama"
}
tensors: [
  { "name": "seq_0", "shape": [2048], "data": [...] },
  { "name": "seq_1", "shape": [1536], "data": [...] },
  ...

training.format.version: string (e.g. "1.0") - Specification version, in case of future changes.

training.dataset.name: string (optional) - Dataset name (e.g. "OpenWebText-ru").

training.dataset.source: string (optional) - URL or description of the data source.

training.file.creation_date: string (ISO 8601) - File creation date.

training.tokenizer.gguf.model: string - Tokenizer model name (llama, gpt2, etc.).

training.tokenizer.gguf.vocab: array[string] - Tokenizer dictionary.

training.tokenizer.gguf.merges: array[string] - Tokenizer merges (for BPE).

training.tokenizer.gguf.pre: string (optional) - Pre-tokenization architecture.

Note: Instead of storing the entire tokenizer, you could reference the model file, but embedding ensures that the data file is completely self-contained.

training.sequence.count: uint64 - Total number of sequences in the file.

training.sequence.lengths: array[uint32] - Key field! An array containing the length of each sequence in tokens. This will allow for efficient "bucketing" (grouping sequences of similar length) in the future.

Naming: training.tensor.{index} (e.g. training.tensor.0, training.tensor.1, ...).

Data type: GGML_TYPE_I32 (standard for tokens in llama.cpp).

Shape: [sequence_length] - One-dimensional array. sequence_length will be different for each tensor.